Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(query/planner): Migrate COPY to new planner #6074

Merged
merged 47 commits into from
Jun 21, 2022

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented Jun 20, 2022

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

This PR will fix #5781.

There are still many works to finish, but opening a draft PR will make it much easier to track my work and give me some guidance.

Details changes

This PR implemented COPT INTO with the new planner.

Most newly added structs and functions are with comments.

Parser and ast

  • Implement with nom
  • Introduce a new struct called CopyTarget

Binder

We used to embed PlanNode in CopyPlan but it doesn't work on the new planner.

So I switch to implementing CopyPlanV2 with some small changes to its layout.

Interpreter

Implement CopyInterpreterV2 so that we don't afraid to break old ones.

Most codes are copied from the old CopyInterpreter.

Task list

  • Add credential support back for URI location
  • Add unit test for parser
  • Enable stateful tests with new planner

Remaining works should be addressed by #6102

Changelog

  • New Feature

Xuanwo added 10 commits June 17, 2022 15:03
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
@vercel
Copy link

vercel bot commented Jun 20, 2022

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Updated
databend ⬜️ Ignored (Inspect) Jun 21, 2022 at 2:01PM (UTC)

@mergify
Copy link
Contributor

mergify bot commented Jun 20, 2022

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

@mergify mergify bot added the pr-feature this PR introduces a new feature to the codebase label Jun 20, 2022
Xuanwo added 7 commits June 20, 2022 16:55
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
…new-planner

Signed-off-by: Xuanwo <github@xuanwo.io>
@Xuanwo
Copy link
Member Author

Xuanwo commented Jun 20, 2022

I'm working on some cool ideas: parsed location at the parser so that we can give users more friendly error reports!

image

Signed-off-by: Xuanwo <github@xuanwo.io>
@Xuanwo
Copy link
Member Author

Xuanwo commented Jun 21, 2022

image

Oh, I like this parser!

Signed-off-by: Xuanwo <github@xuanwo.io>
Xuanwo added 3 commits June 21, 2022 13:12
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
@Xuanwo
Copy link
Member Author

Xuanwo commented Jun 21, 2022

@andylokandy All comments have been addressed, PTAL

Signed-off-by: Xuanwo <github@xuanwo.io>
@BohuTANG
Copy link
Member

The stateful test is great 👍

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
common/ast/src/parser/statement.rs Outdated Show resolved Hide resolved
common/ast/src/parser/statement.rs Outdated Show resolved Hide resolved
common/ast/src/parser/statement.rs Outdated Show resolved Hide resolved
common/ast/src/parser/statement.rs Outdated Show resolved Hide resolved
common/ast/src/parser/statement.rs Outdated Show resolved Hide resolved
common/ast/src/parser/statement.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@leiysky leiysky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM

Xuanwo and others added 9 commits June 21, 2022 18:54
Co-authored-by: Andy Lok <andylokandy@hotmail.com>
Co-authored-by: Andy Lok <andylokandy@hotmail.com>
Co-authored-by: Andy Lok <andylokandy@hotmail.com>
Co-authored-by: Andy Lok <andylokandy@hotmail.com>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: Xuanwo <github@xuanwo.io>
@Xuanwo
Copy link
Member Author

Xuanwo commented Jun 21, 2022

Most of the comments have been addressed. @andylokandy @leiysky PTAL.

@Xuanwo Xuanwo requested review from andylokandy and leiysky June 21, 2022 12:09
@mergify mergify bot merged commit eee788a into databendlabs:main Jun 21, 2022
@Xuanwo Xuanwo deleted the migrate-copy-into-to-new-planner branch June 21, 2022 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
need-review pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate COPY INTO statement to new planner
5 participants